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ABSTRACT 



nature of 
teachers • 



This paper is part of a series of studies about the 
expertise in teaching and the cognitions of effective . 
A summary offered of psychological research on the nature 
of expertise indicates that experts often know more, in a more 
elaborate way, about the subject than novices and organize knowledge 
in a different fashion. It i~5 also noted that tasks presented elicit 
different levels of schema— that is f the same elements .in a task may 
be identified by both experts and novices, but key elements trigger 
deeper, and more elaborate schema for experts than for novices. A ' 
description is given of an evaluation study in which novices 
(observers with two years of classroom experience) and experts 
(teachers) estimated the overlap between the curriculum to which a 
child had been exposed and a criterion test for each child. Four of 
the teachers and three of the observers were further asked to provide 
of their thoughts while doing the overlap task. The 
findings are reviewed, and the qualitative findings are 
detail; Also Included are quotations of subjects' 
the subjects formed exclusions on and predicted the 
potential abilities and expected achievement of the -children . 
Directions for the overlap telst are appended as well as diagrams 
illustrating the subjects' thought processes. (JDh 
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Novice and Expert Knowledge . r , 

• • ; " ' * 

of Individual Student 's Achievement 

This paper is part of a series of studies into the nature • of 

expertise in teaching, more specifically teaching of the primary subjects 

• * * 

(reading and mathematics)- in elementary schools* The research builds on 
the psychological studies of expertise in "areas such -as physics, 
radiology, spatial mapping, social science and chess (Chi &~Glaser, 1982; 
Chi,^ Glaser' & Reea, 1982; Larkin, McDermott, Simon & .Simon/ 1979; 
Lesgold, in press; Simon & Chase, 1973). The research reported here 

contributes to the work on expertise by altering the setting from 

r 

constrained abstracted formalized tasks to a setting which' is natural, 
unstructured, and potentially d> sarnie • The research also builds' on the 
educational studies of teacher planning and decisionmaking among reading, 
social studies, writing, and homeroom teachers (Borko & Niles, 1983; 
Jackson, 1966; * Peterson, Marx & Clark, 1978; Shayelson & / Stern, 1981; 
•Yinger, 1977), often referred to as the study of teachers' cognitions in 
order to distinguish it from research on. instructional practices or 
process-product research. 

The work on the cognitions of effective teachers is an outgrowth of 
research on effective instructional practices, which has supported a basic 
rnodUl of elements that^ influence student academic growth. To review 
briefly, the p^c-test achievement of a group of students is a consequence 
of initial ability and attitude, student learning behaviors and teaching 
behaviors (See Figure I). Student learning behaviors are influenced, in 
turn, % by the student's thought processes, by aspects of instructional 



pacing apd management, and by the background characteristics of the 
student. (Leinhardt', Zigmond & Cooley, .1981). Teachers' classroom 
behaviors are influenced by knowledge of the curriculum and subject 
matter, by pedagogical .theory,' by the agenda that is operating not only 
for that day but also that; week and that particular .block of time, and. by 
knowledge of students. It 'is this latter aspect of knowing, students that 
we are. addressing here. 



Insert Figure 1 Here 




Within the domain t)f teacher cognitions considerable research has 
gone into the description of teacher plans (Jackson, 1966;' Peterson, Marx 
& Clark, 1978; Yinger," 1977), and the relationship of ' those plans to 
actions. There has also been some important work that focuses' on the 
relationships between thoughts and actions (Anderson-Levitt, 1981; Clark 
&' Peterson, 1981; Doyle, 1977; Janesick, 1978; McNair, . 1*78-79;' 
Morine, 1976; Morine & Vallance, 1975; Shavelson, 1981). Recently, 
Morine-Dershimer (1982) has expanded- the focus to include examination of 
the socio- linguistic aspects of teaching. 

■ * -\ ' ' ' 

Psychological research on the nature— of -expettise has been well 

summarized in two recent papers, Lesgold (in press) i'.Chi, Glaser & Rees 

(1982). To recapitulate "their summaries., experts differ' from novices in 

: the following ways: First, experts of ten know mere in a more elaborate 

way about the subject than novices , but more important, they organize that 

knowledge in a different 'fashion. Tasks presented elicit different levels 
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of schema - that is, the same elements in a taste (key words pr phrases) 
may be identified by both experts and novices but those keys. trigger 
deeper, more elaborate schema fol: experts than novices , Novices* may have 
the declarative knowledge needed to solve a problem but do not access it* 
An expert spends time conceptualizing the task, and finding a qualitative^ 
representation. A novice spends less proportional t^me conceptualizing 
th£ problem and is more likely to^ attend to the surface features of the 
task. 

In considering the nature of expertise in teaching we start by 
examining the total teaching context and then focus on a. single aspect. 
Teachers'' cognitions are strongly influenced by something we refr*r to as 
the agenda' for the current operating time block, whether that agenda 
encompassed the month or day .or just that class period. An agenda 
includes, .planning, but lesson plans are only a fragment of the total 
cognitive plan that the teacher brings to the -.lesson presentation. We 
.conceptualize teachers as operating with predefined agendas that .are ■ 
modified* by issues of time and student response. An agenda is* a -dynamic 
plan that operates iike those described by Hayes-Roth & Hayes^R^t]^^978) 
and Sacerdoti (1977). The agenda Is constantly being revised and" updated 
by the teachers' subject matter knowledge, by^ his/her knowledge about 

0 

students, by knowledge about time, and indirectly by ciirricuium knowledge. 
A summary of that conceptualization is presented in Figure 



Insert Figure 2 Here 
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^Figure 2 shows teacher behaviors as , a consequence of teacfher 
cognitions. Teach^T .cognitions are°in turn influenced by the following: 
social* cues from students*'; verbal' dues froja. students; the teacher's 
knowledge of the curriculum and ^subject matter; m the agenda in use; time; 
and the teacher's knowledge of .students (cognftively -and affectively). 
The agenda is in turn affected by curriculum knowledge, time constraints, 
and student knowledge. 

The current study is about midway along a theoretical continuum, of 
distance : from a natural teaching act to an artificial laboratory task. It 
focuses on a known element of importance in a restricted situation, namely 
teacher knowledge of students' potential, for academic success. 

KnowlelQe of students encompasses* several categories of information. 
It includes information .about the student's, home life, siblings, attitudes 
towards school, subjects, peers, etc. Some of our recent evidence 
suggests that teachers are acutely aware of such factors. It also 
includes information about the academic experiences and competencies of 
students/' Previous work indicated that teachers were quite remarkable in 
"""their ability to assess what students know and don' t know,. (Leinhardt & 
jSeewald,- 1981). Specifically, teachers could estimate the fit between 

what had been taught and what would be tested quite accurately* Further, 

■ . • • .» * ■ „ 

the estimate of that fit ia^^a^ significant predictor of student criterion 
performance (Cooley & Leinhardt, 1980;' Leinhardt & Seewald, . 1981).. The 
knowledge- that teachers' views of student competency were of substantive 
interest in and of themselves and ' that a quantitative estimate of 
competency had predictive power in estimating achievement attracted us to 
this aspect of teacher cognitions. - 
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The Subjects " : 

V 

As part of a larger evaluation study (Leinhardt, Zlgmond & Cooiey, 

1981), we ft^ave studied reading Instruction In' classrooms for the learning 

disabled. We^^ser^d in these classrooms for approximately two^ years. 

Observers collected extensive data on student academic progress and .were 

thoroughly familiar with the curricula usfed to teach reading.. The 

observation system required clgse and careful analysis of the specific 
• * m * 

r * * . 

reading actions the students engaged in. There was moderate student turn 
over (55 students out of 105 were in both years of the study) and there 
were approximately 12 students in each class*- sometimedv less, never more. 
Novices consisted of eleven observers who had twenty hours \i observation 
tife firsts year, *and thirty hours of observation in. the" .second year. 
Experts consisted of the teachers in .the classrooms under study* We asked 
all eleven (experts) • teachers and all eleven observers (novices) to 
estimate the overlap between the curriculum to which a chi^d had been 
exposed and a criterion test . for each .child. v 

In addition, fotar of , the teachers and three cf : the observers were 

*■»«<-. 

further asked to provide us with discussions -of their 7 thoughts while\(ioing 

the overlap task. The four teachers had coi^siderable experience teaching 

and were the best of the teachers. Expertise, $or these four teachers, is 

defined in part by the growtjj of the students ^nd in ^ part by skill in 

bringing students into contact &th Appropriate- subject matter (high 

levels of academic engaged time.) The 3 novices were chosen from among the 

observer team: One was , an experienced LD teacher; .one was a former 

» 

teacher, and one had no teaching experience (2 were female; one was 

.* * * t- ■ 

male). * The. novices had been richly exposed to the reading ^behaviors of" 
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the students but had a much lower density -of exposure, and. less reason to 
consider themselves knowledgeable about the .student than teachers, that 
is, they had not been addressing salient instructional questions while 
observing. / ' . t * 

The Task and Methods 



As one measure of assessing teachers' knowledge of students, we 

obtained protocols^of their ^thoughts about students and instruction as 

they were performing the overlap task. The overlap task involves -giving a 

teacher a copy of a standardized achievement test (the, day of tjhe 

administration or shortly thereafter) and asking the teacher to go through 

it and report whether a child has had sufficient instruction to get each 

item correct or not. For this study only the reading subtest was used. 

Vte dtd not ask whether or not the teacher thought the child could get it 

right although many of the- teachers and observers commented on this; 

\ • / " " * 
rather, we emphasized whether the instruction covered had been sufficient 

'for that particular child. Essentially we were asking, the teachers and 

observers to recall instruction and to speculate on its relevance for 

criterion tasks.' This requires the respondents to analyze the ~ criterion 

task into teachable subcomponents and. search*. the known instructional space 

for a match. The organization 3f the instructional space is in and of 

" * i * 

«. t t 

itself -of considerable interest. , In this study each* sub ject estimated the 
level 'of overlap between test and instruction for each child in the class 
(usually 12 students). 'For 2 of the children we asked the subjects to 
think alopd on 20 items, reporting whethei? or not the child had sufficient 
information to pass thaft, item, "and to give their rational^ for their 
decision. The Appendix displays the items afcout which 'they were 
specifically asked to think aloud as well' as a copy of the initial 



A 



instructions. In sum, each teacher and novice was given a copy of the 
^ test and each child's name and was asked to fill out all of the items for 
all of the children in the class. For . particular children and particular 
items, the tea^het^was asked to think aloud. * 

^ The overlap task taps an important skill for teaching, knowledge of 

- - ^ r 



student competency. Teachers who have a good sense of this are it), a 
position to teach more effectively, to target instruction to students, and 
to assign tasks -at a more appropriate 'level iJor students. While we' do not 
knoW whether an individual teacher makes use of the information about 
student consistency, it is clear that the teacher can not use it if s/he 
does not Tiave it. In^ addition to tapping an important teaching skill, the 
overlap task is interesting in another way. In order to answer the 
overlap questions the teacher must call on three relatively distinct" 
bodies of information: student skill, Item requirements or analysis, and 
curriculum sequence «j The teacher can go in eith^r^direction,. calling up 

information about the students ^or about the curriculum and then analyzing 

*- • . ' - * " • 

the problem. , " ' 0 

ft 

Overlap scores were obtained by taking the • total number - of items 



estimated to have been covered, dividing by the number of items on the 
test r and multiplying by 100. " The percentage of hits was obtained^Wby 
scoring each item as an agree/ disagree between specific estimates and 
student performance. The agrees were then ^divided bj£ the totalg (agree + 
disagree)' and multiplied by 100. Protocols were, transcribed and analyzed 
with respect to several themes: overall model of explanation,' clashes o£ 

reasons, used, unit of material referred to, sburces of instruction, 

■ > 

: * • • \ * 

student strategy, higher order reason for outcome prediction, and teacher 
explanation of knowledge source.\ Overall model of explanation refers to 
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the general scheme that seemed .to " be used when explaining stuxiejit 
performance.. Classes of reasons refers to the type of explanation. Unit 
of' material refers to the level at which . atalysis was provided by* the 
teacher (phoneme clusters, word, phrase, meaning, etc.). - Source of 
instruction refers to where or when in texts or instruction the material 
was covered. Student T strategy refers to' the way teachers or observers 
expected students to apprpach the task. Hi gher £rder reason refers to Ja 
more overriding or global explanation for expected outcomes. Finally, 
teacher strategy refers to the general; approach teachers or observers use 
to answer the questions. . 
7 ' ■ ■ 

Findings 



0 . 

4 



The quantitative findings ^i 11 b , e reviewed and then the qualitative 
findings will be discussed. The means and stan4ard deviations 'in hits, 
overlap, and actual percentage correct is presented in' Table 1, alotig with 



the correlations. 



Insert Table 1 Here 



Table 1 indicates that "the overlap estimates of experts and novices 
and the actual performances are quite closely clustered. Both teachers 
and observers estimate a slightly higher overlap between ' instruction and 
performance than appears in actual performance. The accuracy- of the 
experts ^and novices , as revealed by thfe hit scores are likewise quite 
clos§, with' novices being less , than one-half 'a standard deviation lower in 
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accuracy. - Given the detailed level t of exposure to both, students and 
curriculum, this finding is not surprising. 

Thja results show a very modest Correlation between expert and novice 
overlap scores and hits* They also show an interesting pattern in that 
teachers' overlap estimates and hits are related to' actual performance 
more' closely . than noy ices'* Thus, teachers estimate overlap more 
accurately and their hit rate^is^bepter for high achievers' (Perhaps "-"'order 
"children who may have been with them longer). Novices' total overlap 
estimate is not so close and their hits, which are quite close in .numbers 
to the teachers',' are distributed across high and low achievers Jaore 
evenly. t 



Insert Table 2 Here 



Table 1 shows the average number of lines of protocol >for novices and 
experts for each of the three subtests of the CTBS level ^C. For all 3 
subtests the teachers have more lines of protocol than the novices. Thus 
they talk more about the children and tests than* the observer^, but the 
real question, of course, is do they say more? The qualitative analysis 
that follows addresses that question • 

' " . ' - . .* i' ' ' ' 
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Vocabulary • Subtest ^/ 

, • • • 

For the first subtest of the reading test, vocabulary , a wdrd list is 
given to the students ^hd a definition is read orally; the student must" 
select the correct word. In, order to do this, the student must listen* to_ 
the definition, remember it;,' search the word list for the one that fits; 
or think of an answer and try, to match their own answer to the word 'list. 
Teachers and observers tended - to . focus on the correct answer and then 
describe the probability of a student knowing or deciphering it.. Several 
features of the protocols aTe useful to examine in order to understand 
differences and similarities between observers and .teachers. 

Unit . The first of the feiature is the unit used in \ describing the • 
student's exposure tOs the' material in the item. Two of thg novices 
focused exclusively on the entire worii. For example, (Wet) "I think 
Chjerie can read * sight words very well, 11 or (bake) fl I think* she can 
recognize' 'bake' " f « or (heavy) "I think she' s been taught size words". A 
second novice said about the word >ake, "Well, iirst thing it would be 
easier for her than-a^boy because you know, they've made a cake with their 
mother ,.^.lt's a simple word really." or for anoclier child, "I don't think 
he's interested in making a' cake". 

c 

In sharp contrast, teachers focusad on the elements of the word that 
made them decodeable, and in general, assumed comprehension. For example, 
for wet*, "I .think he wduld be fafniliar with the word wfet in ^particular, 
because of the Glass analysis , the et cluster*, or : (bake) "We talked about 
silent e mostly from Ginn materials. We didn't do the ake cluster in 
Glass. That' A s in Ginn, long vowels silent ^e, we did that". One of the 
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observers, looking - more like the teachers, tended to focus "on more of the 
decodeable elements of the target words. However, the novices r comments 
were*much more fragmented and just dealt with the element or followed a 
prompt for more information. Of the seven items in the subtest, each of 
the noviceti gave word level analyses in 71 percent of the cases, while 
teachers gave vowels, vowel consonant clusters, rhyme patterns for 66 
.^percent of the cases. Teachers focused on elements of the task that were 
teachable; observers on more general knowledge. It is important to note 
that given the extended observational time, observers had ample exposure 
to the details of the curriculum material. 

Source * An even more noticeable different between the two groups 
appears in their references to different sources ' from which the child 
could gain the information. Teachers cite: phonics work books, flash 
card drill, spelling,, math x lessons, ^text levels, Glass or wrd family 
drills, and rhyming games. Novices rarely identify ^any ^ source at all. 

When they do, it's with phrases like, "basals spend time on that." This 

----- 

reflects differences in information' level and also differences in the way 
the two groups saw the task. Teachers focused on teachable ^elements and 
on sources of that instruction almost as if an underlying question was, 
11 If you were to teach a student this material, how would you do it?" Or, 
"With what materials?" 

J 
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— " 
Strategy . We also examined the protocols to see if one or another 
student strategy was suggested more commonly. Teachers expected , students 
to use the same strategy that they had been taught: namely to sound Out 
the word, recognize ,it and work it back to meaning. One observer, 
described sounding out strategies in some detail, but the other two 
focused on word meaning and global item characteristics. Novices tended 
not to describe how a child might get the item correct and tended not to 
describe the elements required to get it correct. Thus, novices did not 
generate the information spontaneously and, in the absence of probing for 
it, we do not know if they had a concept of student strategy or not. 

Prediction . For some items a higher order reason for a 
prediction, that is, some information that went beyond item features and 
f ocussed^on^sp^cific knowledge about t£e child or situation was given. 
Again, teachers, when they used these, tended to be more specific than 
observers. Consider, for example, the teacher (gallop) "That's two 
syllables. He'll get v mixed up when he sees 'a-1-^1' and he'll call it 
'awl'./ Every time he sees a- 1-1 he calls it awl. He- can't say al [as in 
pal] when he sees that. And just him seeing tfre two syllables, that'll 
throw him ." The teacher combines an in-depth discussion of the phonics 
with 4p aside about the child.. As opposed to the observer saying, (awful) 

"...Now I'm not sure she knows what "terrible 11 means and awful. She'd 

\ . " , 

probably associate those. I don't think it would be from what she d been 

taught in the class, but just from the way ki4 s around . .^you know... 11 This 

points up a difference in specificity as well. Both teachers and 

observers made gratuitous comments about the level of child they were 

talking about: teachers as an excuse for $fy a student would miss an item 
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taught after a discussion of some detail; observers as an explanation of 

/ • .. ./ 

basic performance. / 



Reading Sentences 9 

The second section of ,the test, dealt with reading sentences and was 
labeled a comprehension section. To answer this portion of the test* the 
child has to read a sentence with a list of words embedded in it and 
select a suitable word to insert from the list of alternates* (S>ee 
Appendix) So a child must decode the sentence, hold in memory the segments 
of the sentence that are given, then decode lists of words, trying each in 
the place, or on reading the sentence, generate the correct word and 
search for it. 

Teachers approached this portion of* the test as a' logical extension 

of the previous section. They did. not mention the switch in modality from 

•j 

listening and reading to all reading. They focused on elements below the 
word level used to analyze the correct word, but made more frequent 
reference to total words than in the vocabulary section. Observers clung 
even closer to the word level analysis than before. As with the 
vocabulary "section, teachers referred to the source of key words 
continuously - * v where the sound or total- word was taught. "Okay. Nimber 
6. The word is visit, and he could have gotten a clu£\ for sounding out 
this word because it contains two small words, is and it, [strategy]. And 

t ■ 

all these words, well, he's seen them everywhere in all facets of his 
work. He should ' have been able to sound out the words using' the NRS 
method and I think this word has been presented , not in spelling, possibly 
in phonics, but I'm sure the word has been used as a vocabulary word on 
the board, and blending, and us writing sentences together ±r\ class." 

- 15 . 
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Observers rarely -referred to instructional sources and when they did, it 
was only to the main text. 

Of more -interest was the experts'' expansion out from key words to 
stem words and their recognition that more • than the key word was needed, 
but not all the words were needed/ For example - (o'clock), "Okay.: I'm 
thinking, It* ts nine • fine, in the ... if he doesn't know mornJLng you 
know, it is nine what? He won't know hour probably. Time he should know. 
Children he should know. If he knew morning — ; it's sort of you can get 
it from the content. So. ..I'm thinking morning more than I am o'clock 
because I think he should be familiar with it." 

Another expert focused more on key words and on the distraction. For 
example, (visit), "I don't think she's had the sound. Yes she has. She 
had it in Book 5. Okay, there's a chance she might get that. She's been 
exposed., to it, so I'll check that ~ai5e' "sTiad" it . We've done a lot of 
rhyming with sit,... But I still think like Tom she'll go for care, they 
recognize care and they'll just snap choose it." Teachers were sensitive 
to both the clues and. confusion posed by the more complex format. .They 
pointed out words that were not necessary to know but whose resistance to 
simple decoding would confuse the student. 

The novices showed differing patterns. One examined the total task 
and focussed on a few words, the key plus a couple of .others, and assessed 
the difficulty level in terms of decoding. J?or example, (music) "No. I 
don't % think that she'd be able "to get very much of this ttem because of 
the word listen, there's a silent *Loud x is an o-u 't^srt comes 

together, I don't think 'that she's had any rules for putting those things 
together. And 'music' is t a twd-syllable word with a funny .. .well it's not 
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actually funny ... u sound." This was the same person who did a more fine 

■ % • . ■ 

grained analysis in the vocabulary subtest. The other two novices stayed 

at a more global level. Otje, the former LD'teacher ,' showed very little 

understanding of the task demands -and gave .short abbreviated^zsspon^es, 

""people clap when they enjoy a. show" - I would say she would know that 

ftom experience." Or. from the other novice, "She understands what 'clap' 

me^ns, and that you do it at the end of something." This novice also had a 

strong sense that the basic strategy used by students after word decoding 

would be elimination, of incorrect responses. Although elimination was 

mentioned once or twice by teachers and se'eis a logical test taking 

strategy - if decoding and trying altefcnatives 'is not hard, it wasn't a 

favored approach. Observers seemed to examine the test as if* they were 

taking it and 1 the reading was a little hard (maybe as with the early 

stages of foreign language study). They attribute rather sophisticated 

test taking skills to Vather weak readers. 

The reading comprehension subtest is seen by bcth observers and 
teachers as, requiring a little something more in the way of skill than the 
vocabulary subtest but not as a totally different task or a major 
expansion. . Teachers seem to retain a stronger sense of the reading nature 
of the task while two of the observers at least focus in the understanding 
- with reading assumed - nature of the task. . 



Reading Comprehens ion Passages 
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1 Tfie last^sectiqn of the test includes several passages, each- of which 

is a couple of paragraphs long. For each passage there are two to four 

^ » ■ * » 

questions that relate \ to the passage. The * two samples chosen ^for 
protocols include a story about a horse named Silver, followed by four 
questions , and a short paragraph about spacemen ±n a rocket going to .the 
moon, followed by three questions. The questions include information 
directly embedded in the passage and so called inference questions. These 

questions require the child^to read and understand a question, remember 

i > ■ 

it, and search the paragraph for the one or two sentences that have the 
information or to read and remember' the basic paragraph, then read the 
questions and recognize or search the paragraph for the correct response. 

The mo3t striking 'thing about the protocols is that in four of the 
eight cases, teachers thought a child would, not be able to answer any of 
the questions or would only get one right; pne observer alsp^felt a child 
would miss all the items. Thus, at least at some level, this subtest i^s 



seen as"\much harder than the other two. 

The teachers and observers also shift unit of focus /and attribute 
success or failure to words, phrases or tfiole questions, with very little 
mention of components of decoding. Interestingly, the mention of sources 
drops considerably. When sources are identified, it usually is in 
reference to the class work but not curriculum material. In one case the 
teacher says, "He is in the beginning of Level 5 in Ginn, and up to this 
point they have not been given very many, I mean, the comprehension is not 
involved at this .point... " as an explanation that the child will be unable 
to do, the entire subtest. 



is 
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When teachers identified a strategy that a student is likely to use, 
they refer to key words * or searching text. For example, "Students, 
themselves have taught each other that [To search froni . question to' key 
words in . a passage] J I didn't realize they were doing that until Robin 
one day confessed to me that she didn' t read her story, and she said, 
v Well you know, Cheryl taught me or told me the way to do it..*is to read 
the question and you look up and can find the answer very easily! %u * 
Clearly, this successful test taking behavior is not viewed as an 
appropriate' in-class behavior. However, several teachers and observers 
expected children who * could not read and understand the passages to try 

this approach. * 

■' . ■' '■ 

In describing the reasons for their predictions, both teachers and 
observers often referred to basic personality as reasons for responses. A 
teacher says, in reference to choosing the best last sentence, "He cannot 
make final judgement^, even though we worked on a little bit of this with 
him... He would just never be able to choose an ending." Or, "I think she 
should. ..she • has enough information to do that, if she doesn't panic by 
the number of sentences that she sees. Paragraphs seem to frighten 
her. ..She's been exposed to it minimally." Or, "but this would be 
overwhelming for Tom. By the time he would finish sounding out maybe the 
first sentence, it just would be impossible for him to retain all of that 
to come back arid answer questions. These three responses also point Jup 
the ambivalence frequently expressed: I taught it but s/ tie doesn't know 
it. Statements of this type made by teachers te(nd^ to focus on items * that 
will be missed. Observers have* very* few global reasons for their 
predictions, especially the negative ones. "He'll be able to pick 
% llttle* out. Surely ■ tie 'd be able to read the first Sentence so he'd just 
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take that right of out: there. I think he. would havje a lot of trouble, 
getting through the rest of the paragraph though so he probably wouldn't 
get any more* right • " 1 

i Both teachers and -observers tend to have less . elaborate thoughts 
about children facing paragraphs' Some teachers knew individual students 
would panic -at the sight but for most children £hey^assumed an attempt 
would be made to get some piece of the ^jt- of questions answered* 
Teachers' responses showed an understanding of ^individual students p and in 
discussing items in this subtest they switched from focusing on the 
cognitive elements of the task- and focused more on the psychological 
■'aspects of the child. . 

Conceptions of the Student, Conceptions « of « ,the Judges 

Teachers did not describe the three subtests as classes of tasks that 
the student approaches. However* in their descriptions of the units and 
sources "and, to some extent the student strategies, "in underlying picture 
emerges.* Teachers consider the first subtest to be test decoding and to 

.■»■■" * ■ " ■■ -* 

be well within the boundaries of word meaning. In other -words, the 
vocabulary test title does : iiot y refer to meaning but to reading level. The 
specifics necessary for decoding have either been taught or not taught. 
Observers focused less on the readability issue and more on the notions of 
meaning and word recognition . and the probability of encountering ^the 
words. , 
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For the second subtest,, select the best word in , brackets, teachers 
began to consider issues of ' meantq|^as well as decoding - but again did 
not overtly describe the parameter' of t;he task. Observers stayed at . the 
whole word level and emphasized the general knowledge portion of the task 
for students* ^ '/ 

In the last subtest both teachers and observers. became more vague in 
their reasoning. The teachers changed focus and concentrated on child 
strategies or the possibility of the child being Nj^sychqd out by one 
paragraph rather than on an analysis of skills needed for. the task. 

Perhaps the teachers are right and gearing up for paragraphs is more than 

. • • v - - . - y & 

half the battle. i ... 

\. ■ 

Insert Figure 3 about here ^ * 



When . teachers are faced, with the Nquery concerning . an item > and a 
child's ability to respond to it, they seem- to call on several stores df 

4 

information. A diagram summarizes this in Figure 3. The teacher . 
considers first the raw nature , of the task, what things are needed tf/ 
perform on the item. While this is not a task* analysis per se, it is 
definitely slanted towards how things are taught. Then^tfiere seems to be 
a quick review of features of the item that woujld scare off the child or^ 
make it: impossible Tfbr. him or her to get it correct. This portion of the 
loop focuses on the skills required for categorizing the problem. The 
teacher then starts to focus on whether or not this particular child lias 
had" sufficient instruction to complete the item. This seems to be done by . 
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bringing two or three segments of information together around the question 
of whether or not the child can do the .task: (a) the location of the 
information in a text (level, chapter, page) or several texts; (b) the 
location of the target child with .reference to that - spot; or., (c) the 
location, of any child at or near that spot and then the location of the 
target child with respect to that child. Occasionally, reference is made 
to several texts that teach the same concept or element. Thus, the 
teachers figuratively roll out a scroll-like curriculum, point to the 
place of the information and search for the child either directly or in 
relationship to other children. The following two quotes from teachers 
seem to support this interpretation. « "No. I don't" think he would be^ 
familiar with that. *We're not familiar with the aw or the suffixes \or 



prefixes. We haven't gotten into that. Vocabulary - no. Tapes, reading 
- no, nothing , ' that I can think of.'" "She hasn't had a-w at .all". - 
( Interviewer) VLet me just ask you a question. ,When you say that she 
hasn't this, how do you remember?". (Teacher) "I know that a-w ^ comes 
aftef where she is. Robin just learned that. That's how I know that. 
(I) So, sometimes you do it relative to other kiefs (T)...I have my mind on 
how v I... ...Yeah on Level 9. And that in relation to what Cheryl and 

Rotfin are studying. I knew I just worked with a-wwith Cheryl and Robin 
and that's why I could say like that ." This latter portion of the diagram 
can be thought of as a curriculum structure node. Bringing the task and 
curriculum elements together permit's '''pf oblem solution. 

When the task demands are not ^directly taught, teachers seemed to 

have less of a model of the task itself and were much less specific about 

. • ■ , s - • 

why a* child could or could not <lo the' item. This was especially true for 
the paragraph section of the test. ^ One notion might be that the teachers 
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were tired of doing the task. However, the teachers were asked to^ do 2 
children and with the beginning part of the second child tended to return 

to the same level of detail. \ * 

, \ 

\ 

\ 

' Insert Figure 4 about\here 



Observers who acted as our informed novices were almost as accurate 

• \ 

■ • ■ \ 

as the teachers but with much less articulated understanding as to why 
they made their assessments. Two of the observers seemed strongly 
influenced by the titles of the three subtests. Tl^us, Vocabulary was word 
knowledge and meaning. One observer saw the vocabulary subtest as 
decoding. Figure 4 diagrams their apparent searches to answer "the 
question. There was a less careful consideration of the ^ask demands and 
more arbitrary search for features that might produce child 'failure. The 
task is described mainly by repeating it rather than analyzing v it . ^The 
curriculum search rarely was described except in the most global terms and 
there seemed to be little knowledge if any about what was presented where 
in Ntexts, or where the children were located. One might think that this, 
was simply an issue of exposure. However, observers collected curriculum 

o ' ■ ■ ■ 

location information- read all the curriculum testts and spent considerable 

. -X; • ' - r 

time entering the wordsT 'and their frequency of usage from the maij 
curricula ihto the 'computer . " They also coded what the children wfcre. doi\ 
- that is the nature of the task. Observers seemed to know less and to 
have a less distinct pattern of. searching for the information that thejr do- 

» ■ y 9 • . ■ 

y ■ ■ , 

have, than did teachers.' . , 
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Implications . %J 

The implications of this research are relevant for our understanding 

of what individuals with similar levels of performance can do with the 

* - 
information. Teachers who are collecting their knowledge in 

instructionally relevant settirig a^p poised to use their in^prmation about 

students - for the teaching of students • Our observers, who had collected 

their information in t instractional settings but not in the context of 

instructing, h&ve the data (how kids will perform) but have little grasp 

of the whys or what to do about it. This raises two issues; one concerns 

the education of new teachers; the other the advance of our understanding 

of teacher cognitions. 

* 

Novice teachers might very well perform less proficiently in the 
overlap ^task 1 than did' the observers. Their .ski^i at observing may be 
clouded by the sheer efforj: of survival the first few years. But, if we 
are to help them inr their survival, it would seem wise to teach them^to 
monitor student growth, and exposure to curriculum content, to assess task, 
demands, and to determine where information Co meet those demands is 
located. Assuming that new teachers are unlikely to know their students 
in advance, a thorough knowledge of several interlocking curricula in 
advance would seem to be advisable. - \. . 

l£ wre review what is known about how novices and experts differ, the 
first and most powerful difference lies" with how the task is conceived. 
Teachers tended to' see each item as consisting of teachable elements which 
had or had not been* taught and to merge that with knowledge about 
f students. Three basic schemas were called up: *one- on task assessment, 
one on * curriculum, and one on students. Novices saw the tasks as. a 
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judgment of item difficulty based on familiarity of words - they rarely if 
ever uped a curriculum schema and relied heavily onia type of child schema 
that was very global. Novices, driven by a goal to respond, developed no 
consistent plan of attack to -the items but moved ° rapidly to a judgment 

7 

call. The -internal representation seemed to be "How hard would this be 
for me?" We can be fairly certain that the novices had the curricular , 
declarative knowledge necessary for more elaborate representations and 
strategies for solutions but were unaware of the relevance. The task 
elicited a different schema for solution, one that was child and self 
focused. They had the declarative knowledge but used the 'wrong' 
procedural knowledge. / , - ' S. 

As part of the research about teachers' thoughts while teaching, this 
study increases' our understanding of one small piece of teachers' ^4 
thoughts: students' exposure to curriculum content. . Many other pieces 
are needed before the picture emerges mote ^clearly. Figures 1^ and 2 map 
out some of the elements and relationships , about which we need to leajrn. 



Specifically we Heed to understand the nature c£ the operating, . 'plans 

that teachers carry with them into the lesson. -The .relationship between 

teachers' subject matter knowledge, cla^feroom instruction and .student 

conceptions of subject matter are also relevant for our understanding of 

the knowledge structures of expert teachers. ^By integrating several - of 

** 

these presumably complex elements of thought we can. begin to work towar<i 
more complete and rigorous model of expertise in teaching. 
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Table^l 

Means, Standard Deviations, and Correlations 
N=-105 



Observers 
Teacher 

Actual % Correct 



Overlap 



54.38 {27.20) 
56.33 (23.01) 
52.51 (21.35) 



Hits 



59.98 (10.67) 
64.30(11.90) 



1. Teacher Overlap 

2. Teacher Hits 

3. Actual Score 

4. Novice Overlap 

5. Novice Hits 



X 



.09 

X 



.67 
.39 

X 



28 



.31; 
.11 
.'39 
X 



.23 
.18 
.26 
.31 
X 
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Table 2 ^ * 

Average Number of Lines of Protocol 

and Ranges by Subtest . 



Observer Teacher 

Vocabulary 

Comprehension: 
Sentences 

Comprehension: 
Passages 



27.33 
(20-39) 

25.50 
(15 -49) 

22.33 
(15-36) 



. 32.14 
(30-53) 

43.00 

(25-63) 

35.20 
(29-40) 
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Define features 




of task 



1 1 




0 



No 



Can the 
child do 
the task? 



Where was the 
material covered 
in each relevant 
curriculum? ' 



Answer: 

p 10; Book 5; 

Green reader 



Or 





Or 


I 




Who is near 
these spots? 


V 

*> 








Answer: A, B, C 

s 



Where is target 
in relation to 
this place? 




Where is target 
in relation to 
A,B,C? 




8eyond — 
may get it 
right 



Not there - 
may get it 
wrong 



. Beyond — 
may get it 
right 



Not there - 
may get it 
wrong - 
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Figure 3. 



Diagram of teachers' sequence of thoughts for overlap task. 



Describe task 



Is child able 
to do this 
sort of thing? 



Have I ever 
seen this 
in class? 



' V 



Figur.e 4. Diagram of observers' sequence of thoughts for overlap task. 



ERIC 



33 



\ 



Appendix 



34 



ERIC 



Directions ' 

** 

For the Overlap Estimate Task . 

We're trying to determine how closely this test measures 
what your students have been taught. To do this, we'd like you 
to estimate which items on the CTBS you have taught to each 
student. This is the level of the. CTBS that (student's name) * 
took. I'd like you to look at the items to determine if you 
think he was taught the information recftrired to get each item 
correct. In considering each item, please take into account the - 
way the Hem is presented as well as the content of the item. 

. ' ' ••< 

Directions for Reading # \ 

Vocabulary Subtest , 

In this subtest, I gave a definition of the word marked, 
and the student had to find the word from four alternatives. For 
example, the task for the first^sample item, 0-1, is to name a pet. 
The correct answer has been marked for you to make it easier for you 
to determined (student's name) has been taught the meaning-word 
relationship, and I'd like you to circle the ittSm^umber if you 
think (student's name) has« been taught the infonrimon required to 
complete the item correctly. Now, when you reach an i\$em that has 
been checked,' please say what you're thinking, that is, try to 
verbal ize your thought processes, your search strategies, or in 

r L ' ' ' . V 
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general how you know whether or not (student's name) was taught 

* ■ . -f 

the information required to get that item correct. In other words, 

^ou don't have to tell jne for each item, just for the ones that, 

are checked c / 

. ■ / ' 

l items for Reading 

Vocabulary Subtest ." •«» * 



0 

0 
0 
0 



wet 
dry 
dirty 
clean 



10 



0 
S 

0 
0 



fry 

bakf 

boil 

birthday 



15 



25 



0 
0 

0 
0 

0 

0 

0 

0* 



saddle 
gallop 
slide 
walk , 

four 
count 
spell 
teach 



20 


P 


big 




0 


light 




0 


' heavy 




• 0 


package 


30 


0 


nice 




0 


ugly 




• 


awful 




0* 


arrive 
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0 
0 
0 



load 
want 
hate 
admire ' 



Directions for Reading " • • 

Comprehension: ; Sentences Subtest \ 

' Okay. Now, go to page -4. The Reading Comprehension Sentences 

subtest is c on pages 4, 5 and 6. The student was supposed to read the 

sentence, and choose the word- that makes the sentence correct. 

(Student's name) worked through this subtest by himself. Once again, 

please circle the item number if you think (student's name) was taught 

the information necessary to correctly -respond to the item, and talk , 

about the items we have checked. 



Items for Reading 

Comprehension: Sentences Subtest 



2 • I 0 hour 0 o'clock 

It is nine 

0 time 0 children 



• in ,the morning. 



k ■ 



Grandmother is coming to < 

0 ■' 



0 hill 
0 care 



• vjsit b 
0 family 



► us. 



8 <5' 
Listen to that loud 



• music. 



0 regular. 



0 plate. 0 smiTe. 



0 hop . 0 dime 

People ^ 

0 clap 0 hand 



> when they enjoy a show. 



*" 0 went 0 jumped 



The dog 



0 bark S brought 



the paper to Dad. 



The.< 



0 buy 0 orbit 



0 rent 0 November 



.must be paid today. 
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Directions for Reading 
Comprehension: Passages Subtest 

Okay, now. This is the Reading Comprehension: Passages subtest, 

and It's on pages 7, 8, and 9. The student was asked to read each 

passage, and then answer the questions that follow 1t. Circle the 

Item number 1f you think (student's name) was taught the Information 

necessary to correctly respond to each question, and again, we've 

marked the ones we want you to talk about. 

V 

Items for "Reading 
Comprehension: Passages Subtest 

Silver was a little pony. One day he was running 1n the hills 
and caught "his foot In 'some rocks. Soon a hungry wildcat came 
toward Silver. Silver began cryjng for his mother. 
; Silver's mother was a big, brave/horse. She ran to help him. 
She kicked at the wllcat and made-Joud noises. Then the wildcat 
ran away. Some cowboys came to help the little pony. 

A * • ' 

6 ' What kind of horse was 
& Silver's mother? 

0 little 

• brave 
0 quiet 
0 afraid , . 

7 Which sentence is the best 
ending for the story? 

0 The cowboys hid. ^ ■ 
0 Silver's mother ran away. 

• Silver was a lucky little pony. 
0 Tfifc wildcat was not hungry. 



8 Which name 1s best for. 
this story? 

0 "T^e Cowboys" 

0 "Mother Wildcat" > 

0 "The Wi 1 dcat ' s /rjend" 

0 "Silver and the Wildcat" 

9 What happened first in 
the story? 

0 The wildcat ran away, 
t Silver got his foot caught. 
0 Some cowboys came to help. 
0 Silver's mother kicked at the 
wildcat. ' 



Two spacemenwere in a rocket. Outside, the astronauts 
could see the stars and the moon, c In back of them was the 
earth. In front of them was the moon. The moon seemed to be 
growing larger and larger as the earth grew smaller. 



16 What were the spacemen 
riding in? 



18 What happened as the 
spacemen traveled? 



0 a car 
0 a train 
0 a rbcket* 
0 an airplane 



0 The moon got shiny. 
0 The moon got darker. 
9 The moon looked larger: 
0 The moon looked smaller, 
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Where were the spacemen 
going? 



D 



0 to the rocket 
0 to the moon 
0 to the earth 
0 to the stars 



